We present high-quality J, H and K photometry of four Small Magellanic Cloud stellar clusters with intermediate ages in the 1-7 Gyr range (namely NGC 339, 361, 416 and 419) . We obtained deep Color-Magnitude Diagrams to study the evolved sequences and providing a detailed census of the Red Giant Branch (RGB), Asymptotic Giant Branch (AGB) and Carbon star populations in each cluster and their contribution to the total cluster light. We find that in the 5-7 Gyr old clusters AGB stars account for ~6 % of the total light in K-band, Carbon stars are lacking and RGB stars account for ~45 % of the total bolometric luminosity. These empirical findings are in good agreement with the theoretical predictions. Finally, we derived photometric metallicities computed by using the properties of the RGB and finding an iron content of [Fe/H]=-1.18, -1.08, -0.99 and -0.96 dex for NGC 339, 361, 416 and 419 respectively.